Beats入门
Beats
Elastic Stack
发布日期:   2020-04-02
文章字数:   2,323
阅读时长:   13 分
阅读次数:  

Beats入门

简介

官方定义

  • Lightweight Data Shipper (轻量级数据传送者)
    • Filebeat # 日志文件
    • Metricbeat # 度量数据
    • Packetbeat # 网络数据
    • Winlogbeat # Windows数据
    • Heartbeat # 健康检查

2020-04-01_234342

Filebeat

轻量型日志采集器

处理流程:

  • 输入Input
  • 处理Filter
  • 输出Output

Prospector(勘探者): 负责监控日志文件信息

Harvester(收割者) :负责把Prospector监控到变化的日志信息发送到Spooler进行处理

知识点:

  • 一个Filebeat可以有多个Prospector
  • 每个日志文件拥有自己的Harvester

2020-04-01_234942

Filebeat Input配置简介

  • yaml语法
  • input_type # 拥有两种类型
    • log # 指定本Prospector监控的是log日志文件
    • stdin # 指定本Prospector监控的是标准输入

例:

# prospectors 为一个数组,Filebeat可以配置多个prospectors
filebeat.prospectors:
    - input_type: log                # 第一个prospector
      paths:
        - c:\programdata\elasticsearch\logs\*

    - input_type: stdin                # 第二个prospector

Filebeat Output配置简介

  • Console
  • Elasticsearch
  • Logstash
  • Kafka
  • Redis
  • File

例1:

# 输出信息到elasticsearch
output.elasticsearch:
    hosts: ["localhost:9200"]
    username: "elastic"
    password: "changeme"

例2:

# 输出信息到控制台
output.console:
    pretty: true        # 表示把输出的json格式化,更方便阅读

Filebeat Filter配置简介

  • Input时处理
    • include_lines # 满足条件读取这一行
    • exclude_lines # 满足条件不读取这一行
    • exclude_files # 满足条件不读取这个文件
  • Output前处理 (– Processor)
    • drop_event # 满足条件直接删除
    • drop_fields # 满足条件直接删除这个字段
    • decode_json_fields # 对满足条件的json字段进行解析
    • include_fields # 满足条件加入这些字段, 或者过滤通过这些字段

例1:

processor:
    - drop_event:         # 当正则匹配就删除这个信息
       when: 
           regexp:
               message: "^DBG:"

例2:

processor:
    - decode_json_fields:
        fields: ["inner"]

例2用来处理形如下信息的信息

{"outer": "value", "inner": "{\"data\": \"value\"}"}

处理后结果为

{"outer": "value", "inner": "{"data": "value"}"}

Filebeat + Elasticsearch Ingest Node

  • Filebeat 缺乏数据转换的能力
  • Elasticsearch Ingest Node
    • 新增的node类型
    • 在数据写入es之前对数据进行处理转换
    • pipeline api (Ingest Node 使用的处理api)

Filebeat Module简介

对于社区常见的需求进行配置封装增加易用性,其中封装了如下常见需求:

  • nginx
  • Apache
  • MySQL
  • ……

封装内容:

  • filebeat.yml配置
  • ingest node pipeline配置
  • Kibana dashboard

这些Module是最佳实践参考, 可以不使用,但其中蕴含的最佳配置是参考的不二人选.

Filebeat 收集 nginx log

  • 通过stdin收集日志
  • 通过console输出结果
  1. 下载Filebeat

​ 下载地址: https://www.elastic.co/cn/

  1. 运行filebeat命令

    其中, 要收集的日志信息形如下:

    [jlc@localhost es]$ head -n 2 ./access.log 
127.0.0.1 - - [27/Jan/2020:23:15:54 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
127.0.0.1 - - [27/Jan/2020:23:38:36 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"

其中,使用的Filebeat配置文件(nginx.yml)内容如下:

   filebeat.prospectors:
       - type: stdin

   output.console:
       pretty: true

解压运行filebeat

   # 解压filebeat
   [jlc@localhost es]$ tar -zxf filebeat-6.1.1-linux-x86_64.tar.gz

   # 运行
   [jlc@localhost filebeat-6.1.1-linux-x86_64]$ head -n 2 ../access.log | ./filebeat -e -c ./nginx.yml

  1. 运行输出结果

    [jlc@localhost filebeat-6.1.1-linux-x86_64]$ head -n 2 ../access.log | ./filebeat -e -c ./nginx.yml 
2020/04/01 16:55:38.579816 beat.go:436: INFO Home path: [/usr/local/es/filebeat-6.1.1-linux-x86_64] Config path: [/usr/local/es/filebeat-6.1.1-linux-x86_64] Data path: [/usr/local/es/filebeat-6.1.1-linux-x86_64/data] Logs path: [/usr/local/es/filebeat-6.1.1-linux-x86_64/logs]
2020/04/01 16:55:38.580153 beat.go:443: INFO Beat UUID: 150db773-661f-4fab-b139-07e0ffc8e768
2020/04/01 16:55:38.580182 beat.go:203: INFO Setup Beat: filebeat; Version: 6.1.1
2020/04/01 16:55:38.580475 metrics.go:23: INFO Metrics logging every 30s
2020/04/01 16:55:38.580689 module.go:76: INFO Beat name: localhost.localdomain
2020/04/01 16:55:38.583042 beat.go:276: INFO filebeat start running.
2020/04/01 16:55:38.583207 registrar.go:71: INFO No registry file found under: /usr/local/es/filebeat-6.1.1-linux-x86_64/data/registry. Creating a new registry file.
2020/04/01 16:55:38.586759 registrar.go:108: INFO Loading registrar data from /usr/local/es/filebeat-6.1.1-linux-x86_64/data/registry
2020/04/01 16:55:38.586816 registrar.go:119: INFO States Loaded from registrar: 0
2020/04/01 16:55:38.586869 filebeat.go:261: WARN Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020/04/01 16:55:38.586889 crawler.go:48: INFO Loading Prospectors: 1
2020/04/01 16:55:38.587132 prospector.go:87: INFO Starting prospector of type: stdin; ID: 16876905907669988323 
2020/04/01 16:55:38.587146 crawler.go:82: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2020/04/01 16:55:38.587232 registrar.go:150: INFO Starting Registrar
2020/04/01 16:55:38.587425 harvester.go:215: INFO Harvester started for file: -
2020/04/01 16:55:38.587716 harvester.go:238: INFO End of file reached: . Closing because close_eof is enabled.
{
  "@timestamp": "2020-04-01T16:55:38.587Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.1.1"
  },
  "offset": 190,
  "message": "127.0.0.1 - - [27/Jan/2020:23:15:54 +0800] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36\"",
  "source": "",
  "prospector": {
    "type": "stdin"
  },
  "beat": {
    "name": "localhost.localdomain",
    "hostname": "localhost.localdomain",
    "version": "6.1.1"
  }
}
{
  "@timestamp": "2020-04-01T16:55:38.587Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.1.1"
  },
  "source": "",
  "offset": 188,
  "message": "127.0.0.1 - - [27/Jan/2020:23:38:36 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36\"",
  "prospector": {
    "type": "stdin"
  },
  "beat": {
    "name": "localhost.localdomain",
    "hostname": "localhost.localdomain",
    "version": "6.1.1"
  }
}
2020/04/01 16:56:08.582776 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30004 beat.memstats.gc_next=4473924 beat.memstats.memory_alloc=2924304 beat.memstats.memory_total=2924304 filebeat.events.active=2 filebeat.events.added=2 filebeat.harvester.closed=1 filebeat.harvester.open_files=-1 filebeat.harvester.running=0 filebeat.harvester.started=1 libbeat.config.module.running=0 libbeat.output.events.acked=2 libbeat.output.events.batches=1 libbeat.output.events.total=2 libbeat.output.type=console libbeat.output.write.bytes=1078 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0 libbeat.pipeline.events.published=2 libbeat.pipeline.events.total=2 libbeat.pipeline.queue.acked=2 registrar.states.current=0 registrar.writes=1

Packetbeat 简介

  • 实时抓取网络包
  • 自动解析应用层协议
    • ICMP (v4 and v6)
    • DNS
    • HTTP
    • Mysql
    • Redis
    • ……
  • Wireshark (Packetbeat 可以当做轻量级的Wireshark)

Packetbeat解析HTTP协议

解析Elasticsearch http请求

  1. 下载Packetbeat

    下载地址: https://www.elastic.co/cn/

  2. 配置配置文件(http.yml)

    packetbeat.interfaces.device: lo   # packetbeat监听的网卡

packetbeat.protocols:               # packetbeat监听的协议
- type: http
  ports: [9200]                    # packetbeat监听的端口
  send_request: true
  include_body_for: ["application/json","x-www-form-urlencoded"]

output.console:
  pretty: true

  3. 运行Packetbeat

    # sudo 抓包时,需要有root权限
# -strict.perms=false Packetbeat运行时需要检查配置文件,加上这句就不会检查了,方便运行

[jlc@localhost packetbeat-6.1.1-linux-x86_64]$ sudo ./packetbeat -e -c http.yml -strict.perms=false

  4. 在浏览器输入请求

    http://localhost:9200/

  5. 控制台得到输出

    {
  "@timestamp": "2020-04-01T17:40:05.307Z",
  "@metadata": {
    "beat": "packetbeat",
    "type": "doc",
    "version": "6.1.1"
  },
  "status": "OK",
  "proc": "",
  "path": "/",
  "query": "GET /",
  "beat": {
    "name": "localhost.localdomain",
    "hostname": "localhost.localdomain",
    "version": "6.1.1"
  },
  "client_port": 54914,
  "port": 9200,
  "server": "localhost.localdomain",
  "type": "http",
  "http": {
    "request": {
      "headers": {
        "content-length": 0
      },
      "params": ""
    },
    "response": {
      "code": 200,
      "phrase": "OK",
      "headers": {
        "content-length": 278,
        "content-type": "application/json; charset=UTF-8"
      },
      "body": "\u001f\ufffd\u0008\u0000\u0000\u0000\u0000\u0000\u0000\u0000|\ufffd[K\ufffd@\u0010\ufffd\ufffd\ufffd+\ufffd\u003c\ufffd!\ufffd\ufffd\ufffd\ufffdc\ufffd\ufffd\ufffd ־ԗ\ufffdI\u0026fd/e/\ufffdP\ufffd\ufffd\ufffdE\ufffdA\ufffdq\ufffd\ufffdo\ufffd\ufffd\\\u0002BB\ufffd%\ufffd$#\ufffd\ufffd5\ufffd\ufffdv\ufffd\ufffd\ufffd\ufffd\ufffd[\u0007\u0026\ufffdUAp밴\ufffdM\ufffdL=\ufffdc5LX?ܭء\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd#\ufffd\ufffd\ufffd\ufffdh\ufffd\u0000cQ\ufffd\ufffdw\ufffd\ufffd~\ufffd\ufffd\u0005\ufffd\u0001\\F4\ufffd\ufffd\ufffd\ufffd\u0017\u001eE\ufffd7\ufffd6\ufffdVT+\u0006i=U+\ufffd\ufffd\ufffd,\ufffd露9M\u000f,\ufffdX\ufffd\ufffdE\ufffd$\ufffd\ufffdS\ufffdU\ufffdd\u001b\ufffdz\ufffd\ufffd\ufffd\ufffd\ufffd*\ufffdKP\ufffd\ufffdd\u000b\ufffd.J\ufffdCKT(\ufffd\ufffd\ufffdh /\ufffd\u003cq\ufffd\u0005\nt\ufffd\ufffdYD˿\u000c\ufffd\n\u003e\ufffd\ufffd\ufffd\u000e\ufffdN\ufffd\u000e\ufffdq\ufffdM\ufffd\u001a\ufffd:jO\ufffdJ\ufffdg\ufffdֆ\ufffd\ufffd/\u000f\ufffd\ufffd\u0017\u0000\u0000\u0000\ufffd\ufffd\u0003\u0000\ufffd+Cɰ\u0001\u0000\u0000"
    }
  },
  "bytes_out": 389,
  "client_server": "localhost.localdomain",
  "client_ip": "::1",
  "bytes_in": 385,
  "client_proc": "",
  "responsetime": 3,
  "request": "GET / HTTP/1.1\r\nHost: localhost:9200\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2\r\nAccept-Encoding: gzip, deflate\r\nConnection: keep-alive\r\nUpgrade-Insecure-Requests: 1\r\nCache-Control: max-age=0\r\n\r\n",
  "ip": "::1",
  "method": "GET"
}
{
  "@timestamp": "2020-04-01T17:40:05.407Z",
  "@metadata": {
    "beat": "packetbeat",
    "type": "doc",
    "version": "6.1.1"
  },
  "bytes_in": 340,
  "client_ip": "::1",
  "status": "OK",
  "proc": "",
  "client_port": 54914,
  "client_proc": "",
  "client_server": "localhost.localdomain",
  "query": "GET /favicon.ico",
  "method": "GET",
  "server": "localhost.localdomain",
  "request": "GET /favicon.ico HTTP/1.1\r\nHost: localhost:9200\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2\r\nAccept-Encoding: gzip, deflate\r\nConnection: keep-alive\r\n\r\n",
  "path": "/favicon.ico",
  "bytes_out": 1652,
  "port": 9200,
  "http": {
    "request": {
      "headers": {
        "content-length": 0
      },
      "params": ""
    },
    "response": {
      "phrase": "OK",
      "headers": {
        "content-length": 1559,
        "content-type": "image/x-icon"
      },
      "code": 200
    }
  },
  "ip": "::1",
  "beat": {
    "name": "localhost.localdomain",
    "hostname": "localhost.localdomain",
    "version": "6.1.1"
  },
  "type": "http",
  "responsetime": 16
}
{
  "@timestamp": "2020-04-01T17:39:59.949Z",
  "@metadata": {
    "beat": "packetbeat",
    "type": "doc",
    "version": "6.1.1"
  },
  "client_proc": "",
  "client_ip": "127.0.0.1",
  "proc": "",
  "responsetime": 7524,
  "query": "GET /_nodes/_local",
  "bytes_out": 519,
  "server": "localhost.localdomain",
  "beat": {
    "name": "localhost.localdomain",
    "hostname": "localhost.localdomain",
    "version": "6.1.1"
  },
  "request": "GET /_nodes/_local?filter_path=nodes.*.settings.tribe HTTP/1.1\r\nHost: localhost:9200\r\nContent-Length: 0\r\nConnection: keep-alive\r\n\r\n",
  "ip": "127.0.0.1",
  "path": "/_nodes/_local",
  "method": "GET",
  "client_port": 48786,
  "http": {
    "response": {
      "body": "HTTP/1.1 200 OK\r\ncontent-type: application/json; charset=UTF-8\r\ncontent-length: 329\r\n\r\n{\"nodes\":{\"Eyro1LLhQOSsyBQNJV86kQ\":{\"ip\":\"127.0.0.1\",\"version\":\"6.1.1\",\"http\":{\"publish_address\":\"127.0.0.1:9200\"}},\"1P40_XHRTKKnKP-covkxLg\":{\"ip\":\"127.0.0.1\",\"version\":\"6.1.1\",\"http\":{\"publish_address\":\"127.0.0.1:7200\"}},\"_d5BzFgWQZGxUcT9KAEFaA\":{\"ip\":\"127.0.0.1\",\"version\":\"6.1.1\",\"http\":{\"publish_address\":\"127.0.0.1:8200\"}}}}HTTP/1.1 200 OK\r",
      "code": 200,
      "phrase": "OK",
      "headers": {
        "content-length": 432,
        "content-type": "application/json; charset=UTF-8"
      }
    },
    "request": {
      "headers": {
        "content-length": 0
      },
      "params": "filter_path=nodes.%2A.settings.tribe"
    }
  },
  "client_server": "localhost.localdomain",
  "port": 9200,
  "bytes_in": 131,
  "type": "http",
  "status": "OK"
}
赏

   转载规则


《Beats入门》 由 Jiavg 采用 知识共享署名 4.0 国际许可协议 进行许可。
  目录